We do not perceive any physical cause of another’s actions but use our understanding of their inner state as a hidden causal variable for their behaviors. This involves a detachment of perceptual similarity from causal similarities so that the mental factors of ToM form a class of hidden variables that we add to our perception to understand causal relations in a social context.
Source: wiktionary